Software manufacturers are engaged in a constant race to bring their software applications to market. The burden to outdo the competition, by providing a product that captures a substantial portion of the market share, bounces back and forth between the various software manufacturers. This fierce competition is further exacerbated by the continuous demand for software applications that support entertainment-based subject matter, media applications, as well as computationally-intensive applications. As a result, the time available to a software manufacturer to bring a new software application to market is steadily decreasing.
As a result, software manufacturers place a significant amount of effort in quality assurance (QA) testing of a product prior to a first customer shipment. Generally, a QA team works with the software developers to identify testing areas for generating as many test cases as possible, with the goal of testing the entire product. One test generated from such test cases is a negative test. As described herein, “a negative test” refers to a test designed to test product behavior in response to error conditions applied to the executing software application code. Unfortunately, the amount of errors or bugs within software application code increases according to a logarithmic scale as a length of the code increases. As a result, it is generally not possible to ensure that software application code is one hundred percent (100%) tested.
A QA team that gives into the pressure to place a software application in the stream of commerce may inadequately test the software product. As a result, consumers of the software product may be dissatisfied with the product due to an unduly amount of errors encountered during usage of the software product. Conversely, QA teams that spend undue amounts of time testing a software application run the risk of failing to capture a portion of the market share by delaying first customer shipments of a product.
Generally, QA teams lack the capability to provide some measurement of how thoroughly an application has been tested. Without such a measurement, QA teams run the risk of either performing inadequate testing of a software application, which eventually results in consumer dissatisfaction with the product due to an undue amount of errors. Conversely, it is possible to go overboard with the testing of a software application, which generally results in a substantial delay in bringing the software application to market.
Furthermore, the complexity of such applications is not diminishing but, in fact, continues to grow as developers attempt to comply with requests or desires to add additional features to such software applications. For example, software applications having four to five million lines to code and multi-layer software applications having hundreds of different layers are not uncommon. As a result, it is nearly impossible to provide a comprehensive test script to test the entirety of such complex software applications to provide some assurance that each module, component or feature of such software is adequately tested.
As a result, in today's modern development environment, the problem of testing complex software applications is often subdivided, such that different components, modules or features are separately tested by the members of a QA or other product testing team. Accordingly, even assuming that the various members of the QA or product testing team adequately test each module, feature or function separately, there is no assurance than the entirety of the product is being tested to provide sufficient quality assurance to enable a first customer shipment. Hence, in this modern development environment, the lack of some capability to provide some measurement of how thoroughly a software application has been tested is exacerbated by the piecemeal testing of the various modules, components or functions of a complex software application.